Effective MSE Criterion for Combined Linear-Viterbi Equalization

Combined linear-Viterbi equalizer (CLVE) is a class of receivers employing a linear pre-filter before the Viterbi algorithm (VA) in order to shape the original channel impulse response to some shorter desired impulse response which allows a lower complexity VA. In this paper, we present a new approach for optimizing CLVEs by making use of the Genetic Algorithm to minimize the effective mean squared error (EMSE). Unlike the conventional MSE approach, the EMSE approach takes into account the effect of noise correlations resulting from the pre-filtering process. Simulation results indicate that the proposed EMSE optimization algorithm has an advantage over the conventional MSE algorithms in that better system performance can be achieved.